API/UCP/UCT: Add per-endpoint qos level - #11805

Summary: contrib/test_jenkins.sh step test_ucm_hooks aborted at iteration 22/30: line 1039: 2154650 Illegal instruction timeout 10 ./test/apps/test_hooks -n 59 → timeout: the monitored command dumped core → exit 1, so the Bash task exited with code 1.

Root cause: SIGILL inside the process while UCM installs mmap hooks by live-patching library code. ucm_bistro_apply_patch_atomic() patches in three steps: (1) atomically write a 2-byte self-jump lock (jmp -2) over the function prologue, (2) spin for ucm_global_opts.bistro_grace_duration wall-clock time hoping any thread already executing past those 2 bytes has left the region, (3) memcpy the remaining patch bytes and then atomically unlock. Step 2 is a pure time-based heuristic with no proof of quiescence. The test intentionally maximizes this race: 59 concurrent threads ((RANDOM % (2*NPROC)) + 1) hammering mmap() while dlopen() triggers hook installation. On a loaded shared CI agent, a thread descheduled mid-prologue can resume after the grace window and execute the half-written instruction stream, which decodes to an invalid opcode → SIGILL. The log corroborates timing-sensitivity: 21 iterations (1–58, 72, 76, 80 threads) passed, and the crash appeared only on one iteration — there is no hang (all iterations complete in ~1.0–1.5 s, no gap >2 min), so this is a genuine race, not a timeout.

Evidence this is unrelated to the PR: the branch ep-traffic-class-api (commit a870f66) touches the UCP/UCT endpoint traffic-class API, while the crash is in src/ucm/bistro code patching, exercised before any UCP endpoint code runs. Git history shows this race is pre-existing and previously band-aided by c77bb48 "UCM/BISTRO: Increase grace time to reduce patch race (#11112)".

Side observation (not the failure, but a latent bug worth filing): earlier in the same log, ./test/apps/test_ucp_config -c 'UCX_MODULES=^ib,rdmacm' printed error: '.../build-test/test/apps/.libs/test_ucp_config' does not exist — the binary was never built. Because test_jenkins.sh:1028 pipes into tee and only checks the subsequent grep, this silent failure is swallowed and the module allow-list check is effectively a no-op.

Implicated commit: unknown for the crash itself (pre-existing race); the most recent related change is c77bb48, shasson5 — "UCM/BISTRO: Increase grace time to reduce patch race (#11112)". PR commit a870f66 is not implicated.

File: src/ucm/bistro/bistro.c:86-97 (time-based grace window + non-atomic memcpy of patch tail); test harness at contrib/test_jenkins.sh:1043

Suggested fix:

1. Immediate: re-run the failed job — this is a flaky, load-dependent race unrelated to PR API/UCP/UCT: Add per-endpoint qos level #11805. Do not block the traffic-class-API PR on it.
2. Short term in CI: make test_ucm_hooks retry a failing iteration (e.g. 2–3 attempts) before declaring failure, and on failure archive the core file plus test_hooks.log so the faulting RIP/symbol is recoverable instead of only "dumped core".
3. Real fix in UCM: replace the wall-clock spin in ucm_bistro_apply_patch_atomic() with a deterministic quiescence barrier — e.g. membarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE) (falling back to the current grace spin when unavailable) so all threads are guaranteed to have serialized their instruction pipeline before the tail memcpy. Additionally, prefer the 5-byte near-jump path (ucm_bistro_jmp_near_patch_t, bistro_x86_64.c:345-358) wherever reachable, since it shrinks the unlocked tail to 3 bytes, and consider scaling bistro_grace_duration with load as a stopgap.
4. Separately: add an explicit exit-status check (or set -o pipefail) around the test_ucp_config invocation at contrib/test_jenkins.sh:1028 and fix the missing build target, so the module allow-list test isn't silently skipped.

Related: PR #11112 ("UCM/BISTRO: Increase grace time to reduce patch race", commit c77bb48) — prior mitigation of this same race; no open issue found matching "test_hooks Illegal instruction".

Summary: make test in build-test/test/gtest exited with Error 1 because exactly one test — test_rwlock.lock — failed, out of 8538 tests run.

Root cause: Evidence from the log: [ PASSED ] 8537 tests. / [ FAILED ] 1 test, listed below: test_rwlock.lock, where TypeParam = and GetParam() =, then make: *** [Makefile:4713: test] Error 1. Nothing else in the ~3697 s run failed, and there is no hang (log timestamps are continuous, longest inter-line gap is ~11 s during cma/uct_p2p_rma_test.put_zcopy). test_rwlock.lock is a pure ucs_rw_spinlock_t unit test (see src/ucs/type/rwlock.h, a self-contained header with no UCP/UCT endpoint dependencies) that synchronizes reader/writer threads with fixed sleeps. On a loaded CI worker those sleeps race — the same suite shows test_rwlock.memory_barriers at 30162 ms, the 14th-longest test in the run, confirming this suite is heavily timing/scheduling dependent. The PR under test (ep-traffic-class-api, commit 168d00c) touches the endpoint traffic-class API and cannot affect the UCS rwlock primitive, so this is a pre-existing flake, not a regression from the PR. The concurrent open PR #11809 ("GTEST/UCS: Increase rwlock test sleep to avoid CI races") confirms this is a known CI race.

Implicated commit: Not the PR commit. The rwlock primitive/test originates from 8bbe776 "UCS: Introduce lightweight rwlock" (Artemy Kovalyov) — the only commit touching src/ucs/type/rwlock.h. No commit in ep-traffic-class-api is implicated.

File: test/gtest/ucs/test_rwlock.cc (the test_rwlock.lock case); primitive under test: src/ucs/type/rwlock.h:56-135

Suggested fix: Treat this as an unrelated flake for PR #11805 — retrigger the Azure "Tests new on worker 0" job, and rebase the branch after PR #11809 (which increases the rwlock test's sleep to remove the CI race) is merged. Longer term, make test_rwlock.lock deterministic instead of sleep-tuned: replace the fixed sleeps with explicit synchronization (a pthread_barrier_t/condition-variable rendezvous, or a polling loop on an atomic flag with a generous timeout) so the assertion no longer depends on thread scheduling latency on a busy worker. Do not raise any job time limit — the run completed and the only failure is this assertion.

Related: #11809 (GTEST/UCS: Increase rwlock test sleep to avoid CI races); failing build PR #11805

Summary: UCP AM test over dc_x on RoCE timed out waiting for the persistent-data receive callback, then crashed with SIGSEGV (make: *** [Makefile:4713: test] Segmentation fault (core dumped)), aborting the entire gtest run.

Root cause:

• Log evidence: test starts at 09:04:07.555, fails at 09:04:17.864 — exactly the ~10 s wait_for_flag polling timeout, i.e. am_data_hold_cb was never invoked, so the 1‑byte eager AM sent over dc_x was never delivered on the receiver. No UCX error/diag was printed, which is the signature of a silently dropped/misrouted IB packet rather than a reported transport failure. (The 09:04:17 → 09:06:11 gap is only libucs backtrace symbolization after the signal, not a hang.)
• Crash amplifier: test/gtest/ucp/test_ucp_am.cc:894 uses EXPECT_TRUE(rx_data != NULL) (non-fatal). Execution continues to line 895, EXPECT_EQ(data, *reinterpret_cast<char*>(rx_data)), dereferencing NULL — matching the backtrace frames testing::internal::CmpHelperEQ<char, char> → ucs::test_base::run() test.cc:463 and "Caught signal 11 … at address (nil)". This turns a single test failure into a crashed test binary and a red build.
• Attribution: this is the "roce" worker and the only failing case is the DC (dc_x) UCP data path; PR API/UCP/UCT: Add per-endpoint qos level #11805 ("API/UCP/UCT: Add per-endpoint traffic class", branch ep-traffic-class-api, successor of UCP/UCT/IB: Add per-endpoint traffic class (DSCP) for RC/RoCE (QOS) #11618 "per-endpoint traffic class (DSCP) for RC/RoCE") changes exactly the AH/QP traffic-class attributes used when building RC/RoCE and DC address handles. A per-EP tclass/DSCP value leaking into the DCI/AH creation path (instead of the iface-level RoCE TC autodetection added in e7b7c866 "UCT/IB: Use global traffic class for RoCEv2 with auto configuration") would produce exactly this symptom: connection established, sends posted, packets never arriving. Note the base-branch source does not contain the traffic-class code, so this part is inferred from the failure locality, not read from the diff.

Implicated commit: [REDACTED:Hex High Entropy String] (PR #11805, branch ep-traffic-class-api) for the delivery failure; the NULL-deref crash pattern predates it (test/gtest/ucp/test_ucp_am.cc, last touched by b6b8531, tomerg-nvidia).

File: test/gtest/ucp/test_ucp_am.cc:893-895 (crash site); per-EP traffic-class/AH-attribute code added by PR #11805 in src/uct/ib/base/ib_iface.c / src/uct/ib/dc/dc_mlx5_ep.c (functional cause)

Suggested fix:

1. Make the test fail safely instead of crashing the suite:

   wait_for_flag(&rx_data);
   ASSERT_TRUE(rx_data != NULL);          /* was EXPECT_TRUE */
   EXPECT_EQ(data, *reinterpret_cast<char*>(rx_data));

   (or if (rx_data == NULL) { request_wait(sptr); return; }) — this alone stops one flaky/failing AM test from segfaulting the whole make test.
2. For the real failure, verify in the PR that when no traffic class is requested on an endpoint, the DC/RoCE path still uses the iface-level default (auto-detected RoCEv2 TC / UCX_IB_TRAFFIC_CLASS) rather than overwriting ah_attr.grh.traffic_class/QP tclass with 0 or an uninitialized value; check the dc_mlx5 DCI AH creation path specifically, since it goes through the same uct_ib_iface_fill_ah_attr_* helpers as RC but is not covered by the RC-focused changes.
3. Re-run with --gtest_filter='dcx/test_ucp_am_nbx.rx_persistent_data*' and UCX_LOG_LEVEL=debug on the RoCE node to confirm reproducibility and dump the effective tclass/DSCP for the DC endpoint; compare against a build of the merge base to rule out an environment flake.

Related: #11805, #11618, commit e7b7c86 "UCT/IB: Use global traffic class for RoCEv2 with auto configuration (#11105)"

Summary: The Azure Pipelines "Codestyle / commit title" job (build 133025) exited with code 1 during its Bash step after validating the PR's commit titles.

Root cause: The commit-title linter iterates over the commits in PR #11805 and requires each title to match UCX's convention <MODULE>[/<SUBMODULE>]: <Capitalized description>. The log shows exactly two evaluations:

• Bad commit title: 'discard change' — no module prefix, not capitalized → rejected
• Good commit title: 'API/UCP/UCT: Add per-endpoint traffic class' → accepted

The Bash task then failed (##[error]Bash exited with code '1'). Nothing was compiled or tested; the merge commit 04c9d0081 (merge of 1fcbbb4b into 45a9372265) checked out fine, and the checkout/fetch output before the error is entirely normal git progress with no timing gaps. So this is a deterministic lint failure, not flakiness, an infra issue, or a hang.

Implicated commit: [REDACTED:Hex High Entropy String] (branch ep-traffic-class-api) is the PR head; the offending title discard change belongs to a commit in that branch's PR range. Not a regression in the CI scripts — the checker behaved as designed.

File: No source file is at fault. The failing step is the "commit title" Bash task of the Codestyle stage in the Azure pipeline definition under buildlib/ (exact path not resolvable from this checkout; the tool could not read buildlib/tools/commit_title.sh, buildlib/commit-title.sh, buildlib/pr/check-commit-format.yml, or buildlib/az-check-commit-title.yml).

Suggested fix: Rewrite the branch history to remove the non-conforming title:

git rebase -i [REDACTED:Hex High Entropy String]
# mark the 'discard change' commit as 'fixup' (or 'reword' it to e.g.
#   'API/UCP/UCT: Fix traffic class ...')
git push --force-with-lease origin ep-traffic-class-api

Simplest option, since discard change looks like a scratch/WIP commit: fixup/squash it into API/UCP/UCT: Add per-endpoint traffic class so the PR carries a single well-formed title. Re-running CI without changing history will fail identically.

Related: none (searches for prior "Bad commit title" reports returned only unrelated PRs: #11838, #11796, #11480, #11218, #2724)

Summary: Azure Pipelines job "UCX PR (Codestyle / commit title)" build 137646 exited with code 1 after codestyle_check_commit_title flagged a commit title.

Root cause: The commit-title checker iterates over all non-merge commits in base..head and requires the title to match the regex ^[0-9A-Z/_-]+:\ [[:alnum:]_]. PR #11805 (branch ep-traffic-class-api) contains three commits; the log shows:

• Good commit title: 'API/UCP/UCT: Rework to abstract QoS level'
• Bad commit title: 'discard change' ← has no MODULE: prefix
• Good commit title: 'API/UCP/UCT: Add per-endpoint traffic class'

The intermediate WIP/fixup commit discard change was never squashed away, so codestyle_check_commit_title returned 1.

Secondary CI-script bug: buildlib/pr/codestyle.yml:18 sets set -eE, so the non-zero return from codestyle_check_commit_title (line 24) aborts the script immediately and the if [[ $? -ne 0 ]] block at lines 25–30 never runs. That's why the log shows only a bare ##[error]Bash exited with code '1' instead of the intended "Bad commit title(s), see " annotation — which makes this failure much harder to self-diagnose from the Azure UI.

Implicated commit: The offending commit is the one titled discard change on branch ep-traffic-class-api (PR head [REDACTED:Hex High Entropy String], merge commit 5b3c4c81a). The checker logic itself is from cfd9cd1f "BUILD/CI: Limit PR commit range checks (#11527)" by Yossi Itigin — it is behaving as designed, not broken.

File: buildlib/tools/codestyle.sh:14 (the title regex) — triggered via buildlib/pr/codestyle.yml:24

Suggested fix:

1. Primary (PR author): rebase the branch and squash or reword the discard change commit so every commit title matches MODULE/SUBMODULE: Capitalized description, e.g. git rebase -i [REDACTED:Hex High Entropy String] and mark it fixup into the relevant API/UCP/UCT: commit, then force-push. Since it's a "discard change" commit, folding it into its parent is almost certainly the right move.
2. Secondary (CI maintainers): make the failure message actually surface. In buildlib/pr/codestyle.yml, either drop -e for that call or capture the status explicitly:

   codestyle_check_commit_title "$base_revision..$head_revision" || {
       url="https://github.com/openucx/ucx/wiki/Guidance-for-contributors#general-guidelines"
       echo "##vso[task.logissue type=error]Bad commit title(s), see $url for more info."
       echo "##vso[task.complete result=Failed;]"
   }

   As written, set -eE makes the existing if [[ $? -ne 0 ]] branch dead code.

Related: PR #11805 (#11805); checker introduced/narrowed by #11527 (#11527); style rules at https://github.com/openucx/ucx/wiki/Guidance-for-contributors#general-guidelines

Summary: Azure job UCX PR (Codestyle / commit title) build 137671 failed: Bad commit title: 'discard change' → ##[error]Bash exited with code '1'.

Root cause: codestyle_check_commit_title iterates the PR's non-merge commits (base^1..head^2) and validates each subject against the regex ^[0-9A-Z/_-]+:\ [[:alnum:]_]. Three commits in branch ep-traffic-class-api passed (API/UCT: Move the QoS level back to the connect parameters, API/UCP/UCT: Rework to abstract QoS level, API/UCP/UCT: Add per-endpoint traffic class), but the commit titled discard change has no AREA: prefix and no capitalized component tag, so the function returned 1 and set -eE in the pipeline step aborted the job immediately. Purely a contributor-side commit-message violation, no product/CI regression.

Implicated commit: The offending commit is the one titled discard change on branch ep-traffic-class-api (PR head [REDACTED:Hex High Entropy String]); the check logic itself is unchanged since cfd9cd1f (Yossi Itigin, "BUILD/CI: Limit PR commit range checks", #11527).

File: buildlib/tools/codestyle.sh:14 (regex), invoked from buildlib/pr/codestyle.yml:24

Suggested fix: Rewrite the PR history to remove/rename the discard change commit, e.g. git rebase -i 22ff0221 and either fixup/squash it into the related API/UCP/UCT: ... commit or reword it to a compliant title such as API/UCP: Revert traffic class field change, then force-push. Secondary CI cleanup (optional): in buildlib/pr/codestyle.yml, the if [[ $? -ne 0 ]] block that prints the wiki link is dead code because set -eE aborts on the non-zero return — change line 24 to codestyle_check_commit_title "$base_revision..$head_revision" || rc=$? (or drop -e around that call) so contributors actually see the "see Guidance-for-contributors" hint instead of a bare exited with code '1'.

Related: Commit title rules: https://github.com/openucx/ucx/wiki/Guidance-for-contributors#general-guidelines; PR under test: #11805; check-range change: #11527